rustc-ap-rustc_lexer 677.0.0

Low-level Rust lexer. The idea with `librustc_lexer` is to make a reusable library, by separating out pure lexing and rustc-specific concerns, like spans, error reporting an interning. So, rustc_lexer operates directly on `&str`, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token. Tokens produced by this lexer are not yet ready for parsing the Rust syntax. For that see [`librustc_parse::lexer`], which converts this basic token stream into wide tokens used by actual parser. The purpose of this crate is to convert raw sources into a labeled sequence of well-known token types, so building an actual Rust token stream will be easier. The main entity of this crate is the [`TokenKind`] enum which represents common lexeme types. [`librustc_parse::lexer`]: ../rustc_parse/lexer/index.html